!--[bse_logo_textminingcourse](https://bse.eu/sites/default/files/bse_logo_small.png)-

Text Mining: Models and Algorithms¶

Problem Set 1¶

Import Functions¶

import nltk
nltk.download('wordnet')
nltk.download('stopwords')
nltk.download('punkt')

True

import pandas as pd 
import numpy as np
import matplotlib.pyplot as plt 
import os
import seaborn as sns
import statsmodels.api as sm

# ML Functions
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split, cross_val_score, KFold
from sklearn.metrics import mean_squared_error

# NLP functions
import nltk
from nltk.stem import WordNetLemmatizer
from nltk.stem import PorterStemmer
lemmatizer = WordNetLemmatizer()
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfVectorizer
from textblob import TextBlob


import spacy
sp = spacy.load('en_core_web_sm')

import re
from nltk.corpus import stopwords

import warnings
# Ignore SettingWithCopyWarning
warnings.filterwarnings("ignore")
warnings.filterwarnings("ignore", category=UserWarning, module="pandas")
warnings.filterwarnings("ignore", category=FutureWarning, module="pandas")

Create DataFrames¶

barcelona_p1 = pd.DataFrame(pd.read_csv('./Barcelona_Data_06-08.csv'))
barcelona_p2 = pd.DataFrame(pd.read_csv('./Barcelona_Data_13-15_06.csv'))
control_p1 = pd.DataFrame(pd.read_csv('./Valencia_06-08.csv'))
control_p2 = pd.DataFrame(pd.read_csv('./Valencia_13-15.csv'))

Dummy variables¶

barcelona_p1['Treatment_period'] = 0
barcelona_p2['Treatment_period'] = 1
control_p1['Treatment_period'] = 0
control_p2['Treatment_period'] = 1

barcelona_p1['Treatment_city'] = 1
barcelona_p2['Treatment_city'] = 1
control_p1['Treatment_city'] = 0
control_p2['Treatment_city'] = 0

Concatenate the Dataframes¶

df = pd.concat([barcelona_p1, barcelona_p2, control_p1, control_p2])

Exploratory Data Analysis¶

# Create a new column for combined treatment condition
df['Treatment_condition'] = df['Treatment_period'] * 2 + df['Treatment_city']
df['Price'] = pd.to_numeric(df['Price'].str.replace('€', '').str.replace('.', ''), errors='coerce')
mapping = {0: 'Control_city_p1', 1: 'Treatment_city_p1', 2:'Control_city_p2', 3:'Treatment_city_p2'}
df['Treatment_city_and_treatment'] = df['Treatment_condition'].map(mapping)
df = df.drop('Treatment_condition', axis = 1)

df

Plotting Prices with the different Treatment conditions¶

• Treated city in treatment Period
• Control city in Treatment period
• Treated city in Control Period
• Control city in Control Period

# Create a boxplot with conditional grouping
plt.figure(figsize=(10, 6))
sns.boxplot(data = df, x ='Treatment_city_and_treatment', y = 'Price')

# Add labels and title
plt.xlabel('Treatment Condition')
plt.ylabel('Prices')
plt.title('Boxplot of Prices by Treatment Condition')

# Show the plot
plt.show()

The plot shows that there is a difference in the distribution of data as the observed prices for the treatment city have a greater standard deviation compared to the values in the control city. It also highlights a difference in the average price in the treatment city in the two different periods which is not matched in the control city, showing that there might be a positive effect of treatment on the prices.

Average Prices¶

average_prices = df.groupby('Treatment_city_and_treatment')['Price'].mean()
average_prices

Treatment_city_and_treatment
Control_city_p1      253.240000
Control_city_p2      258.440000
Treatment_city_p1    374.038462
Treatment_city_p2    437.423077
Name: Price, dtype: float64

Regressions¶

Regression based on city¶

# Regression with only "treatment period" dummy
X1 = sm.add_constant(df['Treatment_city'])
model1 = sm.OLS(df['Price'], X1)
results1 = model1.fit()
print(results1.summary())

                            OLS Regression Results                            
==============================================================================
Dep. Variable:                  Price   R-squared:                       0.135
Model:                            OLS   Adj. R-squared:                  0.135
Method:                 Least Squares   F-statistic:                     499.1
Date:                Wed, 31 Jan 2024   Prob (F-statistic):          7.62e-103
Time:                        17:44:10   Log-Likelihood:                -21196.
No. Observations:                3200   AIC:                         4.240e+04
Df Residuals:                    3198   BIC:                         4.241e+04
Df Model:                           1                                         
Covariance Type:            nonrobust                                         
==================================================================================
                     coef    std err          t      P>|t|      [0.025      0.975]
----------------------------------------------------------------------------------
const            256.1520      5.153     49.709      0.000     246.049     266.255
Treatment_city   147.4659      6.601     22.340      0.000     134.523     160.409
==============================================================================
Omnibus:                     1801.341   Durbin-Watson:                   1.726
Prob(Omnibus):                  0.000   Jarque-Bera (JB):            14393.906
Skew:                           2.615   Prob(JB):                         0.00
Kurtosis:                      11.977   Cond. No.                         2.96
==============================================================================

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Interpretation of regression 1¶

The first regression 1 provides with the average difference in hotel price between Barcelona and Valencia during the period starting from the 13th to the 15th of June 2024. The average difference in hotel price between the 2 cities is 147.5 euros and is significant. This indicates that on average, during that period, average prices in hotels will be 147.5 euros higher in Barcelona than in Valencia. However, it is important to mention that this first regression does not control for other potential confounding factors. Hence, the difference may not necessary originate from the event happening in that period

Regression based on treatment period¶

# Regression with only "treatment period" dummy
X2 = sm.add_constant(df['Treatment_period'])
model2 = sm.OLS(df['Price'], X2)
results2 = model2.fit()
print(results2.summary())

                            OLS Regression Results                            
==============================================================================
Dep. Variable:                  Price   R-squared:                       0.005
Model:                            OLS   Adj. R-squared:                  0.004
Method:                 Least Squares   F-statistic:                     14.85
Date:                Wed, 31 Jan 2024   Prob (F-statistic):           0.000119
Time:                        18:15:14   Log-Likelihood:                -21343.
No. Observations:                3188   AIC:                         4.269e+04
Df Residuals:                    3186   BIC:                         4.270e+04
Df Model:                           1                                         
Covariance Type:            nonrobust                                         
====================================================================================
                       coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------------
const              332.8992      4.925     67.597      0.000     323.243     342.555
Treatment_period    26.7051      6.930      3.854      0.000      13.118      40.293
==============================================================================
Omnibus:                     1823.024   Durbin-Watson:                   1.502
Prob(Omnibus):                  0.000   Jarque-Bera (JB):            14272.722
Skew:                           2.681   Prob(JB):                         0.00
Kurtosis:                      11.871   Cond. No.                         2.63
==============================================================================

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Interpretation of regression 2¶

The second regression indicates the average difference in hotel price between the period starting from the 13th to the 15th of June 2024 and the previous period starting from the 6th to the 8th of June 2024. The average difference in hotel price between the 2 periods is 26.7 euros and is significant. In other words, hotel prices during the period going from the 13th to the 15th appear to be higher in both cities in comparison to the previous period.

Regression Based on city and Treatment Period¶

# Regression with both "treatment period" and "treatment city" dummies and their interaction
df['Interaction_term'] = df['Treatment_period'] * df['Treatment_city']
X3 = sm.add_constant(df[['Treatment_period', 'Treatment_city', 'Interaction_term']])
model3 = sm.OLS(df['Price'], X3)
results3 = model3.fit()
print(results3.summary())

                            OLS Regression Results                            
==============================================================================
Dep. Variable:                  Price   R-squared:                       0.151
Model:                            OLS   Adj. R-squared:                  0.150
Method:                 Least Squares   F-statistic:                     189.4
Date:                Wed, 31 Jan 2024   Prob (F-statistic):          4.83e-113
Time:                        17:44:10   Log-Likelihood:                -21166.
No. Observations:                3200   AIC:                         4.234e+04
Df Residuals:                    3196   BIC:                         4.236e+04
Df Model:                           3                                         
Covariance Type:            nonrobust                                         
====================================================================================
                       coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------------
const              253.2400      7.699     32.893      0.000     238.145     268.335
Treatment_period     5.2000     10.288      0.505      0.613     -14.972      25.372
Treatment_city     120.7985      9.519     12.690      0.000     102.134     139.463
Interaction_term    58.1846     13.153      4.424      0.000      32.395      83.975
==============================================================================
Omnibus:                     1778.169   Durbin-Watson:                   1.758
Prob(Omnibus):                  0.000   Jarque-Bera (JB):            14105.793
Skew:                           2.574   Prob(JB):                         0.00
Kurtosis:                      11.905   Cond. No.                         7.90
==============================================================================

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Interpretation of regression 3¶

In the third regression, we include an interaction term and both city and period dummy variables. The coefficient 'Interaction Term' represents the difference-in-differences (DiD) estimate.

It captures the differential effect of the treatment on prices by considering the interaction between "treatment period" and "treatment city" and allows us to tell whether the effect of the festival period on hotel prices is significantly different in Barcelona compared to Valencia.. The DiD method helps to control for time-invariant factors and isolates the impact of the treatment by comparing the changes in the treatment city to the changes in the control city over time.

The need for a second city (control city) is crucial in the DiD analysis as it allows us to have counterfactual. The control city accounts for trends or factors affecting both the treatment and control groups similarly. The interaction term (Treatment_period × Treatment_city) captures the differential effect by considering how the treatment effect varies across the treatment and control groups.

Since the results show that there is a high, positive and significant coefficient this shows that there is a substantially positive effect of treatment on prices. In other words, we can conclude that the festival SONORA happening in Barcelona during that period does positively influence hotel prices.

NLP - Analyzing Descriptions¶

Utils¶

def abbr_or_lower(word):
    if re.match('([A-Z]+[a-z]*){2,}', word):
        return word
    else:
        return word.lower()

df.shape

(3200, 10)

df = df.dropna()

Preprocessing:¶

• Removing special characters and numbers.
• Converting words to lowercase.
• Removing stopwords.
• Applying stemming.

text_column = 'Descriptions'

# Remove special characters and numbers
# df['processed_text'] = df[text_column].apply(lambda x: re.sub(r'[^a-zA-Z\s]', '', x))

# OR (if you want to keep numbers)
df['processed_text'] = df[text_column].apply(lambda x: re.sub(r'\W', ' ', x))

# Convert each word using abbr_or_lower function
df['processed_text'] = df['processed_text'].apply(lambda x: ' '.join(abbr_or_lower(word) for word in x.split()))

# Remove stopwords
stop_words = set(stopwords.words('spanish'))
df['processed_text'] = df['processed_text'].apply(lambda x: ' '.join([word for word in x.split() if word not in stop_words]))

# Apply stemming
stemmer = PorterStemmer()
df['processed_text'] = df['processed_text'].apply(lambda x: ' '.join([stemmer.stem(word) for word in x.split()]))

print(df['processed_text'])

0      mayerl bisb urquinaona ofrec wifi gratuita pre...
1      hotel diseño vista playa barceloneta centro ba...
2      pensión encuentra detrá plaza sant jaum barrio...
3      hotel encantador situado poco minuto pie rambl...
4      h rega adult onli ofrec habitacion air acondic...
                             ...                        
695    hotel goza ubicación idónea junto ciudad art c...
696    sorolla palac ofrec alojamiento elegant ubicad...
697    hotel diseño ilunion aqua 3 situado valencia s...
698    hotel turia encuentra junto estación autobus v...
699    one shot colón 46 ubicado 10 minuto pie catedr...
Name: processed_text, Length: 3188, dtype: object

Creating a document-term matrix (DTM)¶

# Create document term matrix using CountVectorizer

pd.set_option('display.max_columns', None)
vectorizer = CountVectorizer(ngram_range = (1,2), min_df=0.05, max_df=0.30)
X = vectorizer.fit_transform(df['processed_text'])

# Convert DTM to DataFrame
dtm_df = pd.DataFrame(X.toarray(), columns=vectorizer.get_feature_names_out())
display(dtm_df)
dtm_df.shape

(3188, 422)

Percentage of presence of chosen word¶

chosen_word = 'desayuno'
print(f"Percentage of presence of the chose word: {round((dtm_df[chosen_word].sum()/dtm_df.shape[0]),3)*100}%")

Percentage of presence of the chose word: 32.2%

Vizualization¶

Word Cloud¶

from wordcloud import WordCloud
import matplotlib.pyplot as plt

# Create a high-resolution word cloud object with relative scaling
wordcloud = WordCloud(
    background_color='white', 
    colormap='Blues', 
    width=800, 
    height=400,
    relative_scaling= 0  # Adjust the relative scaling value as desired
).generate_from_frequencies(dtm_df.sum())

# Plot the word cloud with a higher resolution
plt.figure(figsize=(10, 6), dpi=300)
plt.imshow(wordcloud, interpolation='bilinear')
plt.axis('off')
plt.show()

Creating Term Frequency (TF)-Inverse Document Frequency (IDF) and text_length features¶

# Create TF-IDF vectorizer
tfidf_vectorizer = TfidfVectorizer(ngram_range=(1, 2), min_df=0.05, max_df=0.30)
X_tfidf = tfidf_vectorizer.fit_transform(df['processed_text'])

# Convert TF-IDF to DataFrame
tfidf_df = pd.DataFrame(X_tfidf.toarray(), columns=tfidf_vectorizer.get_feature_names_out())

# Reset index of the original DataFrame to avoid duplicate indices
df = df.reset_index(drop=True)

# Add length feature
tfidf_df['text_length'] = df['processed_text'].apply(len)

# Concatenate the TF-IDF features and length with your original DataFrame
final_df = pd.concat([df, tfidf_df], axis=1)

# Display the final DataFrame
display(final_df)